Supplement for Where to Add Actions in Human-in-the-Loop Reinforcement Learning

UCRL This strongly optimistic approach is based on UCRL2 (Jaksch, Ortner, and Auer 2010), which defines a confidence set over each transition/reward distribution, and takes the maximum valid distribution in the confidence set when planning. Specifically, in a finite-horizon setting it allows the L1 norm of the transition distribution to deviate from the MLE by at most √ 14 log(SA`τ`/δ) max(N,1) , where ` is the number of episodes, N is the number of transition samples, and δ is a user-specified confidence parameter. UCRL’s bound incorporates global uncertainty to ensure that the true MDP is within the confidence set with high probability (Auer and Ortner 2007). We use this bound to quantify the uncertainty over our outcome distribution, setting δ = 0.05 as in Osband et al. 2013.1. The advantage of UCRL2’s constraint on the L1 is that it is easy to calculate the most optimistic distribution that obeys the constraint, as explained in Strehl and Littman 2004. Since we are in a finite horizon setting, we calculate the most optimistic distribution for each (s, t) pair. MBIE Model-based Interval Estimation (MBIE) (Strehl and Littman 2004; 2008) is a very similar idea to UCRL, but simply bounds the local L1 divergence at each state with probability 1− δ. The original MBIE algorithm (Strehl and Littman 2004) used a bound on their transition dy-